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NONTECHNICAL  SUMMARY 


The  following  model  is  considered:  A  machine  is  inspected  at  the 
beginning  of  discrete  time  periods  and  its  operating  state  (condition) 
is  determined.  After  each  inspection  a  decision  upon  whether  or  not 
to  replace  the  machine  at  the  end  of  the  period  must  be  made.  If  the 
machine  is  replaced  a  new  machine  must  be  purchased.  It  is  assumed 
that  it  takes  one  period  to  purchase  and  install  a  new  machine  —  during 
this  period  the  process  is  said  to  be  in  state  0.  If  the  machine  is 
in  operating  state  (condition)  i  and  a  decision  not  to  replace  is 
made  then  there  are  known  transition  probabilities  F  such  that  the 
operating  state  of  the  machine  at  time  t+1  will  be  j  with  probability 
.  Thus,  for  instance,  F^  is  the  probability  that  a  machine  in 
operating  state  i  will  fail.  If  the  machine  does  fail  during  a  period 
then  it  is  removed  at  the  end  of  the  period  and  a  new  machine  must  then 
be  purchased.  As  before  this  corresponds  to  being  in  state  0. 

It  is  supposed  that  each  time  a  machine  is  classified  as  being  in 
operating  state  i  an  operating  cost  C(i)  is  incurred.  Also  when¬ 
ever  the  process  is  in  state  0  a  cost  C(0)  is  incurred.  This  may 
include  both  the  costs  due  to  buying,  delivering,  and  installing  a  new 
machine  and  also  a  cost  due  to  the  fact  that  no  machine  is  in  use  for 
one  period. 

A  policy  is  any  rule  for  deciding  when  to  replace  the  machine  and 
when  to  leave  it  alone.  It  is  shown  that  if  (l)  C(i)  -  the  operating 

co6t  associated  with  state  i  -  is  increasing  in  i  (for  i  >  0)  and 
if  (2)  the  transition  into  any  higher  block  of  states  (K,K+1, ...}  is 


more  likely  for  a  higher  numbered  state  then  for  a  lower  numbered  state. 
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then  the  policy  which  leads  to  the  smallest  (long-run)  average  cost 
has  a  very  simple  form.  Its  form  is  that  for  some  integer  J  it 
replaces  when  in  operating  state  i  if  and  only  if  i  >  J . 

Methods  for  finding  the  optimal  policy  (or  equivalently  the  critical 
value  j)  are  discussed  and  a  numerical  example  is  given. 

Often  in  practice  there  is  an  additional  cost  incurred  whenever 
a  non-planned  replacement  -  i.e.,  a  replacement  caused  by  a  failure  - 
occurs.  It  is  shown  that  even  in  this  case  the  optimal  policy  has  the 
same  form  as  before. 

In  the  last  section  of  this  paper  the  case  where  more  than  one 
machine  may  be  purchased  at  a  given  time  is  considered.  Thus  when  a 
machine  fails  many  machines  may  be  purchased.  One  could  then  be  put 
in  use  and  the  others  held  as  reserves.  This  might  be  desirable  for 
(l)  it  might  be  less  expensive  to  buy  the  units  in  quantity,  (2)  it 
might  cut  down  on  delivery  and  installation  costs,  and  (3)  it  woulO 
cut  down  the  number  of  periods  during  which  no  machine  is  in  uBe.  It 
is  then  shown  that  a  multi-dimensional  generalization  of  the  optimal 
policy  for  the  original  (no  reserves)  problem  is  optimal  for  this 
problem. 
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A  MARKOVIAN  REPLACEMENT  MODEL  WITH  A 
GENERALIZATION  TO  INCLUDE  STOCKING 
by 

Sheldon  M.  Ross 

1.  Model  and  Summary  of  Results. 

This  paper  is  concerned  with  the  following  countable  state  Markovian 
Replacement  Model:  A  unit  (piece  of  equipment,  system,  etc.)  is  observed 
at  the  beginning  of  discrete  time  periods  t  =  0,1,2,...  and  classified 
as  being  in  one  of  a  countable  number  of  states  labeled  by  the  non¬ 
negative  integers.  After  observing  the  state  of  the  unit  the  observer 
must  choose  one  of  two  possible  actions:  Action  1  is  to  leave  the  unit 
in  service;  Action  0  is  to  remove  the  unit  from  service  at  the  end  of 
the  period.  It  is  assumed  that  a  unit  in  state  i  will  fail  during 
the  period  before  the  next  observation  with  probability  F^.  If  a 
unit  fails  it  is  removed  at  the  end  of  the  period.  Whenever  a  unit  is 
removed  from  service  (either  by  action  0  or  by  action  1  and  a  subsequent 
failure)  a  new  unit  must  be  purchased.  This  corresponds  to  being  in 
state  0.  Thus  only  one  action  is  possible  when  in  state  0  and  that  is 
to  purchase  a  new  unit.  It  is  assumed  that  it  takes  one  time  period 
to  purchase  and  install  a  new  unit. 

If  action  1  is  chosen  at  time  t  then  there  are  known  tran¬ 
sition  probabilities  1=1,2,...  J=0,1, ...  such  that 

p(xt.i-Jlxt-1’At'll'pij  where 

=  state  .of  the  unit  in  use  at  time  t,  and 

A  =  action  chosen  at  time  t. 
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Of  course,  if  action  0  is  chosen  at  time  t  then  X  ,  =  0  - 

t+1 

i  •  e  • , 

p(Xt+i  =  0|Xt  -  i,  A  «  0)  ■  1  for  1=1,2,... 

Also,  when  a  new  unit  is  purchased  there  is  a  kr.  vn  probability 
distribution  (P^)"^  over  its  initial  state  -  i.e.,  ?£Xt+^=i|Xt=o)=P. . 

Thus,  for  instance,  Pq  may  be  Interpreted  as  the  probability  that  a 
new  machine  will  be  inoperative. 

We  shall  call  X^  the  state  of  the  process  at  time  t.  Each 
time  the  process  is  in  state  i  an  expected  cost  C(i)  is  incurred. 

Thus,  for  i  >  0,  C(i)  may  be  interpreted  as  the  expected  operating 
cost  incurred  during  one  period  by  a  unit  which  is  in  state  i  at  the 
beginning  of  the  period.  C(0)  includes  the  cost  of  buying  and  instal¬ 
ling  a  new  unit;  it  may  also  include  a  cost  due  to  the  fact  that  no 
unit  is  in  use  for  that  period. 

A  policy  is  any  rule  for  choosing  actions.  In  sections  3  and  4 
of  this  paper  under  suitable  conditions  on  the  costs  and  transition 
probabilities  (given  in  section  2),  the  structure  of  an  optimal  policy 
with  respect  to  (l)  the  diecounted-cost  and  (2)  the  average-cost  criterion 
is  determined.  Theorems  3.2  and  4.2  of  these  sections  generalize  results 
given  in  [2].  This  generalization  is  in  two  directions.  Firstly,  we 
allow  for  a  countable  number  of  states  (versus  a  finite  number  in  [2]) 
and  secondly  we  allow  for  a  somewhat  more  general  class  of  transition 
probabilities.  The  other  theorems  in  sections  3  and  4  are  new  and 
further  characterize  the  structure  of  the  optimal  policy.  These  theorems 
show  that  the  decision  upon  whether  or  not  to  replace  at  time  t  may 
sometimes  be  determined  solely  by  the  conditional  expected  cost  incurred 


at  time  t+1  given  that  you  don't  replace  and  a  failure  doesn't  occur. 
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For  example,  it  is  shown  that,  under  pure  deterioration,  the  optimal 
average-cost  policy  replaces  only  when  this  conditional  expected  cost 
is  larger  than  the  optimal  average  cost  per  unit  time. 

In  section  5  methods  for  finding  the  optimal  policy  by  exploiting 
its  known  structure  are  suggested.  A  numerical  example  is  given. 

In  section  6  it  is  shown  how  the  previous  results  may  be  extended 
to  the  case  where  there  is  a  penalty  cost  incurred  whenever  a  non- 
planned  replacement  (i.e.,  a  replacement  caused  by  a  failure)  occurs. 

In  section  7  the  case  where  more  than  one  unit  may  be  purchased 
u  5ivcii  wiiuc  xw  couidxU£j.'cd  xv  x>v  wuvn  that  optimal  policies 
exist  and  are  analogous  in  structure  to  those  of  sections  3  and  4. 

This  is  a  joint  stocking  and  replacement  model  and  the  results  generalize 
those  given  in  [3]. 
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2.  Conditions  and  Preliminary  Lemma 

We  impose  the  following  conditions  on  the  costs  and  transition 
probabilities: 

Condition  1:  (C(i)}”_^  is  a  non-decreasing  bounded  sequence 

00 

Condition  2 :  {P^}  ^  is  a  non-decreasing  sequence 

00 

Condition  3-  For  each  k  =  1,2,...  the  function  r.  (i)  =  l/l-P  Z  P 

R  i°J=k  IJ 

is  a  non-decreasing  function  of  i  for  i  =  1,2,...  . 

(where  o/o  is  taken  to  be  °o)  . 

Thus  Conditions  1  and  2  say  that  the  operating  cost  and  failure 
probability  are  both  non-decreasing  functions  of  the  state.  Condition  3 
says  that  the  conditional  probability  of  a  transition  into  any  block  of 
states  (k,k  +1,...),  given  that  action  1  is  chosen  and  a  failure  does 
not  occur,  is  a  non- decreasing  function  of  the  present  state  i  (for  i  >  0) . 

For  notational  ease  we  shall  assume  throughout  that  P  <  1  for  all 
i.  It  is  quite  easy  to  show  that  all  of  the  results  still  hold  even  if 
this  is  not  the  case. 

The  following  lemma  will  be  needed.  Its  proof  may  be  found  in  [2] 

Lemma  2.1:  Condition  3  implies  that  for  every  non-decreasing  bounded 
sequence  (h(j)}°°  1  the  function  k(i)  =  l/l-P..  Z  P.  ,h(j)  is  also  non- 

J=1  lVo  iJ 

decreasing  for  i  =  1,2,...  . 

3.  Discounted-Cost  Solution 

We  are  interested  in  finding  a  policy  R.  such  that 

P 

t(i,P,R.)  =  min  \|r(i,P,R)  for  all  i  =  0,1,...,  where 
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and  where  0  <  0  <  1.  Such  a  policy  is  said  to  be  an  optimal  p-discount 
policy  or  for  short,  a  p-optimal  policy. 

Let  Rp  be  a  p-optimal  policy  and  let  V^(i)  *  ijr(i,S,R^)  i  ■  0,1...  . 


Lemma  3.1:  Under  Conditions  1,  2,  3  fV„(i)  -  C(i)}?  .  is  a 
"  P  1=1 

non-decreasing  sequence. 


Proof:  Let  Vp(i,l)  =  C(i)  for  i  =■  0,1,...  and  define  recursively 


(1)  V  (i,n)  =  min(C(i) +  0Z  P  V (j,n-l);C(i) +  pV„(0,n-l)}  for  i>0 

P  j=0  iJ  P  P 

and 

00 

Vfl(0,n)  =  c(0)  +  3  E  P.V_(j,n-l) 

p  yo  i  0 


We  first  show  by  induction  that  (Vp(i,n)  -  C(i))"=1  is  a  non- decreasing 
sequence  for  each  n.  For  n  ®  1  it  follows  trivially.  Assume  then  that 
(Vp(i,n-1)  -  C(i)]"=1  is  non-decreasing  and  let  0  <  i  <  k.  There  are 
two  cases: 


Case  1:  Vp(i,n)  =  C(i)  +  P  Vp(0,n-l),  which  implies  by  (l)  that 
00 

(2)  E  P  V_(j,n-1)  >V  (0,n-l),  or  equivalently  that 

J=0  p  p 

(3)  1/1-P10  E  P  V  (j,n-l)  >  V _(0,n-l). 

10  J/O  p  0 

Now  (Vp(i,n-l)  -  C(i)}*=1  nondecreasing  implies  that 
.  .  00 

{Vp(i,n-l))i=1  is  nondecreasing,  and  so  by  Lemma  2.1  and  (3) 


k  >  i  >  0  implies 
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(4)  l/l-Pk0  I  Vp(j,n-l)  >Vp(o,n-l),  which  implies  by 
(l)  that 

(5)  Vp(k,n)  -  C(k)  =  0  Vp(0,n-1) 

=  Vp(i,n)  -  C(i)  . 

00 

Case  2:  Vft(i,n)  =  C(i)  +  0  I  P  VA(j,n-l) 

P  J=0  iJ  P 

and  so  by  (l)  we  have  that 


(6)  l/l-P.n  I  P.  a  V _(j,n-l)  <  V  (0,n-l),  now 
10  J/O  iJ  P  3 


(7)  j!0  Puvp(j’n'l)  =  pioV°’n-1)  +  (i-p»)^P10j*)P1Jvf(J.-D 

+  *PkO"Pio'1/1'PiOjP0PijVJ’n'1' 


and  so  by  Lemma  2.1  and  (6)  we  have  that 


(8)  J!0Ve(J’n-1)  s  Vp'0’”-1)  * 

+  (pko-pio>vp(0'n-1) 


=  E  P.  VR(j,n-l)  ,  and  so 
J-0  P 


(9)  V  (i,n)  -  C(i)  <01  P  V  (J,n-1) 
•  P  J=0  J  P 

but  from  (l)  we  also  have  that 


(10)  Vp(i,n)  -  C(i)  <  (3  Vp(0,n-1) 

and  so  from  (9),  (10)  and  (l)  we  get  that 

(11)  Vp(i,n)  -  C(i)  <Vp(k,n)  -  C(k) 

and  so  k  >  i  >0  implies  that  Vp(k,n)  -  C(k)  >Vp(i,n)  -  C(i). 
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by  induction  (V^(i,n)  -  C(i)}"ssl  is  non-decreasing  for  all  n. 

Now  V0(i.n)  is  the  minimum  expected  discounted  costs  incurred  over 
P 

n- stages  given  that  you  start  in  state  i.  Since  0  <  1  and  costs  are 
bounded,  it  is  easy  to  see  that  V^(i,n)  -*Vp(i)  as  n  -»  *>  for  each  i. 

(Vp(i)  -  C(i)}”_^  is  non-decreasing. 

Q.E.D. 

Remark:  Since  (C(i))T  .  is  non-decreasing  it  follows  that  {Va(i)}”  , 

1=1  p  1=1 

is  non- decreasing. 

Definition:  A  policy  R  that  replaces  (takes  action  0)  at  time  t  iff 
Xt  >  J  for  some  J  =  1,2,...  <*>  is  called  a  control-limit  policy.  The 
control- limit  policy  with  J  =  »  is  the  policy  which  never  replaces. 


Theorem  :  Under  Conditions  1,  2,  3  there  is  a  control-limit  policy 
which  is  0- optimal. 


Proof:  It  is  well  known  (see  [ 1 ] )  that  V„(i)  i  =  0,1,2,  ...  satisfies 

00 

V  (i)  =  min(C(i)  +  0  £  P  V(j);  C(i)  +  M„(0))  for  1  =  1,2,.. 
p  j=o  iJ  p  p 

and  any  policy  which  chooses  action  1  in  state  i  when  the  first  term  is 

the  minimum  and  action  0  when  the  second  term  is  minimum  is  0-optimal. 

00 

Let  L  =  *  if  IP  V  (J)  <  V„(o)  for  all  i  >  0,  otherwise 
p  J=0  0  P  P 

00 

let  ip  =  min(i:  I  P^  Vp(j)  >  Vp(c)} 


=  min[ i:l/l-P  £  P  V  (j)  >  V  (o)J 
i°  ij  P  P 
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The  next  theorem  further  characterizes  the  strujture  of  the  optimal  policy. 

Theorem  Under  Conditions  1,  2,  3  if  i  is  such  that 

l/l-P, n  L  P  0(3)  <  (1-8)  V  (0)  (i  >0) 

10  j/0  1J  @ 

then  there  is  a  ^-optimal  control-limit  policy  which  does  not  replace  when 
the  process  is  in  state  i.  Also,  if  the  above  inequality  is  strict  then 
any  policy  which  replaces  at  state  i  is  not  P-optimal. 

Proof:  V.(i)  <  C(i)  +  p  VQ(0)  for  i  *  1,  2,  .. 

P  ~  P 

l/l-P  L  P  V  (j)  <  l/l-P  L  P  C(J)  +  P  V  (0) 

10  j/0  ^  e  i0  j/0  P 

Now  suppose  that  l/l-P,-  2  P,  ,  C(j)  <  (l-P)  V_(o) 

10  J/0  P 

1/:L-pio pu  Vj)  5  V0) 
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L  P  V(j)<V(0) 
j-0  1J  e  e 

and  so  the  first  part  of  the  theorem  follows  from  Corollary  3*3  • 


To  prove  the  second  part  suppose  that 


1/l-P  I  P  C( J)  <  (1-3)  V  (0) 
10  J^O  P 

Then,  as  above,  this  implies  that 


£  PnVa(j)<V(0) 

J=0  iJ  p  3 

Now  let  R  be  any  policy  which  replace  at  state  i 
*r(i,0,R)  =  C(i)  +  0*(O,P,R) 


>  c(i)  +  0  vp(0) 


>  c(i)  +  0  Z  P  V  (J)  >  V  (i) 
J=0  p  p 


and  so  R  is  not  0- optimal. 

Q  •  E.D. 


Often  we  deal  with  a  process  in  whcih  the  unit  in  use  can  only  deteriorate 
in  time.  This  is  represented  mathematically  by  the  following: 

Definition:  If  P^j  =  0  for  all  0  <  J  <  i  then  we  call  the  process  a 
pure  deterioration  process. 

Theorem  3«1<  •  In  a  pure  deterioration  process,  under  Conditions  1,  2,  3, 

the  control-limit  policy  which  replaces  at  those  states  i  for  which 

l/l-P  I  P  C(J)  >(1-0)  V.(0)  is  0- optimal. 

10  J#  1J  P 
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Proof:  Suppose  that  l/l-P..  E  P  C(j)  >  (l-p)  VR(o),  suppose  further 

u  J^O  J  p 

that  there  is  an  optimal  policy  which  does  not  replace  at  state  i 


V  (i)  =  C(i)  +  P  E  P  V  (J) 

P  J=0  iJ  P 

=  C(i)  +  p  P1Q  Vp(0)  +  P  E  C( J )  +  p  E  P<4(Vft(j)-C(j)J 


ito 


iJ 


J/O 


ij  P 


>  C(i)  +  P  P10  vp(0)  +  S(l-Pi0)(l-3)Vp(0)  +  P  E  Pij(Vp(j)'C(j)) 
>C(i)  +  p  P10vp(o)  +  p(i-Pio)(i-p)vp(o)  +  P(l-Pi0)(Vp(i)-C(l)) 


where  the  last  inequality  follows  from  Lemma  3.1. 

/.  (i  -  p  +  p  P10)vp(i)  >  (i  -  P  +  P  p10)(c(i)  +  P  vp(o)) 

or  Vp(i)  >  C(i)  +  P  V  (0) 

which  is  a  contradiction,  and  so  every  optimal  policy  replaces  at  state  i 
and  the  result  follows  from  Theorem  3-4  • 

Q.E.D. 


ThuB,  in  a  pure  deterioration  process,  the  P-optimal  policy  replaces  at 
time  t  whenever  the  conditional  expected  cost  at  time  t  +  1,  given  that 

you  don't  replace  and  a  failure  doesn't  occur,  is  greater  than  (l-P)VQ(o). 

p 

The  significance  and  intuitive  content  of  Theorems  3*4  and  3.5  become 
clearer  when  we  consider  them  in  connection  with  the  average- cost  criterion. 


4.  Average- Cost  Solution 

For  the  average- cost  criterion  we  are  interested  in  a  policy  R* 

such  that  cp(i,R*)  -  min  <p(i,R)  for  all  i  =  0,1,2,... 

R 

n 

where  <p(i,R)  =  lira  E  ER[C(Xt)|x  =  ij/n  . 

n  -» 00  t=0 
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Such  a  policy  R*,  if  it  exists,  is  said  to  be  optimal. 

Wo  shall  need  to  assume  the  following: 

Condition  4:  a  =  minlpQjP^)  >  0. 

In  order  to  prove  the  analogues  of  Theorem  3.2,  3.k,  and  3*5  we  define 
the  following  process.  Consider  a  new  process  (the  prime  process)  with 
identical  state  and  action  spaces,  with  the  same  cost  structure,  but 
with  transition  probabilities  now  given  by 


for  J  k  0 

for  j  =  0 

for  j  f  0 

for  j  «  0 


for  all 


J 


0,1, . .;  i  >  0 


We  shall  make  use  of  the  following  result  given  by  Ross  (see  [5]): 

The  optimal  1-0!  discount  rule  for  the  prime  process  is  also  an  optimal 
policy  (in  the  average- cost  sense)  for  the  original  process;  and 
g  =  a  Vi_Q(0)  where  V^^O)  is  the  optimal  discounted-costs  for  the 
prime  process  and  g  is  the  optimal  average-cost  for  the  original 
process.  -  i.e., 

V'  „(0)  =  min  *'(0,l-a,R') 
i_a  R* 

g  =  min  cp(i,R)  for  all  i  =  0,1,... 

R 

The  following  lemma  is  needed  and  its  proof  is  immediate. 
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Lemma  4.1:  If  Conditions  1,  2,  3  hold  for  the  original  process  then  they 
hold  for  the  prime  process. 

The  following  analogue  of  Theorem  3 *2  is  thus  Immediate. 

o 

Theor«mjK2:  Under  Conditions  1,  2,  3*  4  there  is  a  control-limit  policy 

which  is  optimal  in  the  average- cost  sense. 

Also  noting  that  l/l-P'  £  P'  C( J )  =  l/l-P. n  £  P  C(j) 

u  j/0  u 

and  (l  -  (l-a))V'  (0)  =  a  V'  (o)  =  g  we  get  the  following  analogues 

of  Theorems  3.4  and  3*5* 

Theorem  4.3:  Under  Conditions  1,  2,  3>  4  if  i  is  such  that 

lA-pio  }%  pu  0(J)  2  8 

then  there  is  an  average-cost  optimal  control- limit  policy 
which  does  not  replace  at  state  i.  Also  if  Fi  >  0  and 
the  inequality  is  strict  ther.  any  policy  which  replaces 
whenever  ir.  state  i  is  not  optimal. 


Theorem  4.  4  Under  Conditions  1,  2,  3,  4  if  P^  =  0  for  all  0  <  J  <  i 
i  =  1,2,..  (pure  deterioration)  then  the  control-limit  policy  which 
replaces  at  state  i  if  and  only  if 


l/l-P  £  P 

10  j/o 


C(J)  >  g 


is  optimal  in  the  average-cost  sense. 


Thus  Theorem  4 .3  may  be  interpreted  as  saying  that  if  the  conditional 
expected  cost  for  the  next  stage,  given  that  we  don't  replace  and  a 
failure  doesn't  occur,  is  no  larger  than  the  optimal  average  cost  then 
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we  should  not  replace.  Theorem  4. 4  says  that,  under  pure  deterioration, 
this  is  the  only  time  we  should  not  replace. 


5.  Computation  of  Optimal  Policy 

We  have  shown  that  under  Conditions  1,  2,  3,  4  the  optimal  (discount 
or  average-cost)  policy  exists  and  has  a  simple  structure.  In  thiB  sec¬ 
tion  we  discuss  possible  methods  for  calculating  the  optimal  policy. 

We  have  shown  in  section  4  that  any  method  of  solution  for  the 
discounted- co st  problem  is  also  a  method  for  the  average-cost  problem. 
This  is  so  because  the  average  cost  problem  can  be  converted  into  a 
discounted- cost  problem  by  defining  the  prime  transition  probabilities 


P10  -  g 
1  -  a 


J  =  0 


and  P^j  = 


1  -  a 


V  a 


J  4  0 


J  =  0 


where  a  =  min{P0,P10). 

The  optimal  1  -  a  discount  policy  for  the  prime  process  is  the  optimal 
average-cost  policy  for  the  original  process. 

Similarly,  any  method  of  solution  for  the  average- cost  problem  is 
also  a  method  for  the  discounted- cost  problem.  We  show  this  by  defining 
a  new  proces  (the  star  process)  with  identical  state,  action,  and  cost 
spaces  but  with  transition  probabilities  now  given  by 


J  4  0 


*1  j  =  \  and  PJ- 

J  I  1  -  p  +  P  P1(J  j  =  0  J  j  1  -  P  +  P  PQ  j  =  0 


J  t  0 


Then  it  Is  easy  to  see  that  Conditions  1,  2,  3,  4  are  satisfied  in  the 
star  process  and  it  again  follows  from  Ross'  result  (see  [5])  that 
qp*(0,R)  =  (l-P)i|r(0,P,R)  for  any  (stationary  deterministic)  policy  R. 

Thus  the  optimal  average-cost  policy  for  the  star  process  is  a 
P-optimal  discount  policy  for  the  original  process  if  this  process  starts 
in  sxnte  0.  However,  it  is  easy  to  see  that  this  policy  will,  in  most 
cases,  be  P-optimal  independent  of  the  initial  state.  This  is  so,  for 
example,  if  all  states  communicate  (P^,  >  0  for  all  i  is  sufficient). 

Thus  any  general  method  for  solving  the  discount-coBt  problem  may  be 
regarded  as  a  general  method  for  solving  the  average- cost  problem  and 
vice  versa  (assuming,  of  course.  Conditions  1,  2,  3,  4). 

We  shall  now  discuss  possible  approaches  for  determining  the  optimal 
policy.  The  first  two  will  be  discussed  in  the  discounted- cost  framework 
and  the  last  in  the  average- cost  framework. 

Method  1:  Policy- improvement  Method  (P-discount) 

We  initially  choose  a  policy  R  and  calculate  i|r(i,{3,R)  for  each 
i  =  0,1..  .  We  then  "improve"  R  by  forming  a  new  policy  R  which 
which  takes  action  1  at  state  i  if 
00 

Z  P . .  'KJjPjR)  <  i|r(0,P,R)  and  action  0  if 
J=0 
00 

£  P41  *(J,M)  >  *(0,p,R) 

J=0  J 

Then  it  can  be  shown  that  \|r(i,P,R)  <  i|r(i,P,R)  for  all  i  =  0,1,..  . 
We  then  improve  R,  etc.  When  no  further  improvement  can  be  made 
we  have  the  optimal  policy. 


14 


To  take  into  account  the  structure  of  the  optimal  policy  it  is  sug¬ 
gested  that  the  initial  R  be  chosen  to  be  a  control-limit  policy. 
However,  though  one  would  hope  that  the  improved  policies  would  also 
be  control- limit  policies,  this  is  now  nc  cessarily  the  case. 

Method  2:  Successive  Approximations  (^-discount) 

Let  B(l)  =  space  of  all  bounded  functions  on  the  non-negative 
Integers. 

Define  the  operator  T;  B(l)  -»B(l)  by 

00 

(TU)  (i)  =  C(i)  +  P  min{U(0),  £  P  U(j)}  i>0 

J=0 

00 

(TU)(0)  =  C(0)  +  P  £  P.  U(j) 

J=0  J 

Then  VQ  =  {V_(i))°°  .  is  the  unique  function  such  that 
P  p  1=0 

TV„  =  VQ:  also  lim  T1^  =  VQ  for  any  U  (under  the  supremum  norm). 

P  P  P 

Thus  may  be  obtained  by  successively  applying  the  operator  T 

to  any  initial  vector  U.  The  optimal  policy  may  then  be  gotten  by 
applying  Corollary  3»3« 

In  order  to  take  advantage  of  the  known  structure  of  (Lemma  3.3  ) 

it  is  suggested  that  the  initial  U-vector  has  the  property  that 
(i)  {U(i))”_1  is  a  non- decree  sing  sequence 

00 

(ii)  U(0)  =  C(o)  +  p  £  P  U(  j ) 

J=0  0 

It  is  easy  to  check  that  T11!!  also  has  property  (i). 

Method  3J  Analytic  Method  (average- cost) 

We  assume  that  the  process  starts  in  state  0  and  let  T  =  time  at 
which  the  process  returns  to  state  0 
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4' mm  w 


i.e.  T  =  min{t:t  >  0,  X  -  0} 

Then  it  is  well  known  that  for  any  (stationary)  policy  R 


9(0, R) 


Ej  ri1  C<Xt)|X0-0 

ut»0  J 


V 


when  EpT  is  finite  by  Condition  4. 


Let  *  control- limit  policy  which  replaces  whenever  in  states 


i,  i  +  1, 


•  •  •  • 


Then  g  *  min  9(R)  =  min 

R  i=l,2, . .» 


Go  c(x‘)|x°'°] 


E_  T 
Ri 


and  it  is  sometimes  analytically  possible  to  determine  the  above 
minimum  and  the  minimal  value  of  i.  One  possible  approach,  which 
is  sometime  applicable,  is  to  treat  <p(R^)  as  a  continuous  function 
of  i  and  try  to  minimize  it  by  using  differential  calculus. 


Example .  5-1:  Consider  a  process  for  which  a  unit  in  state  i,  if  not 
replaced,  either  remains  in  state  i  or  falls.  Also  Buppose  that  the 
failure  probability  is  independent  of  the  state  and  also  equals  the 
probability  that  a  new  unit  is  defective. 

Pli  ’  1  '  “  pio  "  a  1  =  1’2'" 


po  =  a 


Then  9^)  = 


i-1  00 

C(0)  +  1/a  L  P  C(J)  +  L  P  C(J) 

_ ,1=1  J  .1  =  1  J 

i-1  00 

1  +  l/a  £  p .  +  L  P 
0-1  J  J-l  0 


1 6 


•  •  • 


Now  suppose  C(j)  =  N(l-(l/2)^)  J  =  1,2, 

•  Pj  =  (l-a)(l/2)J  J  =  1,2, .. . 

.  • .  <p(R  )  .  c(0)^Ni-a/a[l-(l/2)1'1-l/3n/3(lA)1~1]^N(l-a)[(l/2)1-1-(l/j)(lA)1~1] 
1  1  +  l-a/a[l-(l/2)i_1]  +  (l-a)(l/2)1_1 

To  find  the  value  of  i  which  minimizes  the  above  we  differentiate,  set 
equal  to  zero,  and  solve  for  i. 

For  N  =  100,  C(o)  =  200,  a  *  .1.  The  minimum  value  of  i  lies  between 

2  and  3.  Thus  it  seems  reasonable  that  the  optimal  policy  should  be 

either  R0  or  R.,. 

2  3 

To  check  we  first  convert  to  the  discount  (prime)  problem. 

Pj_i  =  1  i  =  1>2>  •  •  • 

Pi  =  V1'0  =  d/2)1  1  =  1,2,... 

p  =  1-a  =  .9 

then  \|r'(0,P,R2)  =  l/a  <p(R2)  =  111 
♦•(1,P,R2)  =  500 

T|r'(i,P,R2)  =  lOOd-dA)1)  +  .9  t'(0,P,R2)  i  =  2,3,... 

=  699.6  +  iood-d/2)1) 

The  improved  rule  is  the  one  which  replaces  at  state  i  if  and  only  if 

V(i,0,R2)  >t'(0,p,R2) 

the  improved  rule  is  R^. 
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Now  y'(0,Mj)  =  i/v  <P(R5)  =  770 
*'(1^,R5)  -  500 
♦'(2,0,R3)  =  750 

\|r'(i,0,R,)  =  lOOCl-Cl/s)1)  +  .9(770) 

i  *  5^>  •  •  • 

=  695  +  100(1- (l/2)  ) 

. " .  the  improved  rule  i s  again  • 

is  the  optimal  average-cost  policy  -  i.e.,  the  optimal  policy 
replaces  at  states  5^,5,...  “  and  the  optimal  average  cost  =  77- 

Q  #  E  -  D  • 


Example  5-2:  By  making  use  of  the  "star”  process  and  the  idea  of  method 
3  we  can  sometimes  get  a  closed-form  expression  for  the  expected  dis¬ 
counted  costs.  Suppose  we  have  (as  in  example  5«l) 


P  =  1-a  ,  P.~  =  a  i  =  1>2, . . 
ii  10 


P  =  a 
0 


Let  Pf4  =  P(l-a)  ,  PJ0  -  1-0  +  & 


ii 
P*  =  3P 


P*  =  1-0  +  00 


i  "“i 

then  ♦(O,0,Ri)-  1/1-P  9*(0,Ri) 


where  <P*(o>Ri) 


i-1  "  ,  . 

C(o)  +  l/a*  L  P*  C(J)  +  £  P?  C(J) 

_ ,1=1  3  - - 

-  ill 

1  +  l/a*  L  P*+  L  P* 

J=1  0  J=i  J 


and  where  or*  =  1  -  0  +  • 


6.  Extension  to  Penalty  Costs 

One  possible  extension  of  the  above  theory  is  to  include  a  penalty 
cost  A  which  is  incurred  whenever  the  process  goes  to  state  0  without 
the  replacement  action  being  chosen  -  i.e.  whenever  a  r.on-planned  replace¬ 
ment  occurs  This  cun  be  treated  by  letting  the  coe;  function  depend  not 
only  on  the  state  but  also  on  the  action  chosen. 

ihus  let  C(i,j)  =  expected  cost  incurred  when  the  process  is  in  state  i 

and  action  j  is  chosen,  i  >  0,  J  =  0,1. 

However,  the  value  of  C(i,j)  depends  upon  whether  we  are  in  the  dis¬ 
count  or  average-cost  case. 

Discount  Case:  In  the  discount  case 

C(i,l)  -  C(i)  +  0  A  P1Q  i  >  0 

C(i,0)  =  C(i) 

This  is  so  since  the  expected  cost  of  not  replacing  when  in  state  i 
includes  the  operating  cost  plus  an  expected  penalty  cost  which  is  dis¬ 
counted  since  it  is  incurred  at  the  next  stage.  It  can  be  shown,  in 
exactly  the  same  manner  as  before,  that  Theorem  J.2  remains  true.  Sim¬ 
ilarly  Theorems  3.4  and  3' 5  can  be  shown  to  remain  true  under  the  proviso 
that 

l/lP,  [  £  P, .  C(j)  +  A  P,  J  replaces 
10  j/0  iJ  i0 

l/l-P.n  £  P. ,  C( j)  in  the  statement  of  these  theorems. 

j/0  1J 

Average-cost  Case:  In  the  average-cost  case 

C(i,l)  =  C(i)  +APi0  i  >0 
C(i,0)  =  C(i)  i  >  0  . 
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It  can  then  be  shown  that  Theorem  4.2  remains  true  and  that  Theorems  4.3 

and  4.4  remain  true  under  the  proviso  that  l/l-P.n  [  £  P  C(j)  +  A  P  ] 

0  j/0  10 

replaces  l/l-P.rt  £  P  C(j)  in  the  statement  of  these  theorems.  This 
10  J/O  iJ 

is  shown,  as  before,  by  reduction  to  the  discount  case  via  the  prime  prob¬ 
lem.  Theorems  4. 3  and  4.4  follow,  for  instance,  from  the  discounted 
results  by  noticing 


that  C(i,l)  =  C(i)  +  A  P1Q 

=  C(i)  +  A((l-a)P^0  +  a) 
=  C(i)  +  A' (l-a)P|Q 


where  A'  =  A 


-u;rio 

(l-a)P! 


+  a- 

and  P^  =  < 

|  1  -  a 

>1 

10  - 

P._  -  a 
iO 

t  1  -  a 

J  i  0 


j  =  0 


The  results  then  follow  by  further  noticing  that 


pij  o(j)  *a,iv  -  ih"p10  ^  pu  c<j>  ♦  A,vi-pi0 


■  lA-Pio[^0  pu  c(j)tA  V  • 
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7 •  Generalization  to  Include:  Stocking  of  Reserve  Units. 

It  has  been  assumed  up  to  now  that  each  time  a  unit  fails  or  is 
replaced  a  new  unit  must  he  purchased  and  a  cost  C(0)  is  incurred. 

This  cost  C(0)  includes  both  the  cost  of  buying,  delivering,  and  in¬ 
stalling  the  new  unit  and  also  a  cost  due  to  the  fact  that  no  machine 
is  in  use  for  a  period,  We  shall  now,  however,  allow  for  the  possibility 
of  purchasing  more  than  one  unit  and  keeping  some  in  reserve.  This 
might  be  desirable  for  (l)  it  might  be  less  expensive  to  buy  the  units 
in  quantity,  (2)  if  might  cut  down  on  delivery  and  installation  costs, 
and  (3)  it  would  cut  down  the  cost  due  to  idleness  (no  units  in  use). 

The  state  space  will  consist  of  states  0  and  (n,i)  i=l,2, . . .n=0,l, . . .W 

The  process  will  be  said  to  be  in  state  (n, i)  when  the  unit  in  use 
is  in  operating  state  i  and  there  are  n  units  in  reserve.  State  0 
corresponds  to  the  situation  where  there  are  no  units  in  reserve  and 
no  (operating)  unit  in  use.  If  the  process  is  in  state  (n,i)  at  time 
t  then  one  of  two  possible  actions  must  be  chosen.  Action  1  leaves 
the  unit  in  use  alone  and  action  0  replaces  it.  If  the  unit  in  use 
is  in  operating  state  1  and  acrion  1  is  chosen  then  the  unit  will 
fail  before  the  next  observation  with  probability  If  the  unit 

fails,  and  reserve  units  are  on  hand,  then  the  unit  will  be  replaced 
at  the  end  of  the  period.  When  the  process  is  in  state  0  there  are  N 

possible  actions  -  a  ,...,a  -  where  action  a  corresponds  to  pur- 

L  N  K. 

chasing  K  new  units.  Whenever  a  new  (or  reserve)  unit  is  put  in  use 

.  ,00 

then  we  again  assume  that  there  is  a  probability  distribution  IP  J  ^ 
over  its  initial  state.  Howe\er,  we  shall  now  suppose  that  there  is 
zero  probability  of  a  new  unit  being  (initially)  inoperative.  This 
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condition  is  not  essential  but  it  will  simplify  notation  and  it  also 
clarifies  the  above  formulation.  The  transition  probabilities  will 
thus  be  as  follows: 

P(Xt+1  -  (n,j)|Xt  -  (n,i),  .  l]  -  P  0  <  n  <  K-l  i,J  >  0 

P{Xt+1  «  (n-l,J)|Xt  =  (n,i),  At  =  l)  =  PiQP  1  <  n  <  N-l  i,J  >  0 

P(ltt+1  -  °lxt  *  (O'1’'  At  -  11  -  pi0  1  >  0 

P(Xt+]  =  (n-l,j)|Xt  =  (n,i),  At  =  0)  =  Pj  1  <  n  <  N-l  i,J  >  0 

P[Xt+1  =  0|Xt  =  (0,i),  At  =  0}  =  1  i  >  0 

P(Xt+1  =  (K-l,j)|Xt  =  0,  At  =  aK)  =  Pj  1  <  K  <  N-l,  J  >  0 
00  00 

where  T  P  -  1,  T  P  =  1  i  >  0. 

J-l  J  J-0  J 

The  costs  are  as  follows:  Whenever  the  process  is  in  state  (n, i)  an 
expected  cost  C(n,i)  is  incurred.  C(n,i)  includes  the  expected 
operating  cost  of  a  unit  in  state  i  and  the  inventory  costs  involved 
in  holding  n  units  in  reserve.  When  the  process  is  in  state  0  and 
action  is  chosen  then  there  is  a  cost  C (K )  incurred  -  this  in¬ 

cludes  the  cost  of  buying,  installing,  and  delivering  K  new  units; 
it  may  also  include  a  cost  due  to  no  unit  being  in  use  for  a  period. 

We  shall  now  determine  the  structure  of  the  optimal  policy.  In 
the  P-discount  case  the  results  follow  in  an  almost  identical  manner  as 
in  section  J.  However  for  the  average-cost  case  the  method  of  section  4 
no  longer  works  and  a  new  method  of  attack  is  developed. 
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Consider  the  following  condition; 

Condition  l1;  (c(n,i))"_^  is  a  non-decreasing  bounded  sequence  for 


each  n  =  0,1, . . .,N-1. 

“  t 

For  any  policy  R,  let  iKZ,P,R)  =  I  P  EptCfX^A^)  |xq  =■  Z] 

t=0 

where  Z  =  0,  (n,i)  i  >  0,  n  <  N-lj  and  where 
C(X  )  for  X  £  0 

C(X  A)  =  ]  1  .Let  V„(Z)  =  min  t(Z,P,R). 

(C(K)  for  Xt  =  0,  At  =  ak  P  R 

Lemma  7-1-  Under  Conditions  l',2,5  (V^(n,i)  -  CCn,!))”^  is  a  non¬ 

decreasing  sequence  for  each  n  =  0,1,...,N-1. 

Proof:  Same  as  proof  of  Lemma  J.l.  Q.E.D. 

Definition:  A  policy  R  is  said  to  be  a  generalized  control-limit 
policy  if  there  exists  integers  (possibly  infinite)  iQ, ...,iN  ^  and 
an  integer  K,  1  <  K  <  N,  such  that  R  cnooses  action  0  when  in 
state  (n,i)  iff  i  >  i^  and  R  chooses  action  a^  whenever  in 
state  0. 

Theorem  7.2:  Under  Conditions  1 ' ,2,3  there  is  a  generalized  control- 
limit  policy  which  is  S-optimal. 


Proof: 


Vp(n,i) 


C(n,i)  +  P 

=  min  / 

C(n,i)  +  d 


r  ve(n,j)  +  p  P  r-  P  V  (n-l,j) 

j/o  w  j=l  J  p 


r  P  Vs(n-l,j) 
j=l  J 


'The  argument  now  follows  as  presented  in  theorem  5.2.  Q.E.D. 

Thus  the  (3 -optimal  policy  purchases  units  when  in  state  0,  and 

when  there  are  n  units  in  reserve  it  replaces  the  unit  in  use  if  its 
operating  state  is  larger  than  some  preassigned  number  i  -1. 
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For  any  Z  =  C,  (n,i)  i  >  0,  n  <  N-l,  let 

m 

<P(Z,R)  =  lim  E  E  f  C ( X  ,A  )  | X  =  Z] 
m  -» oo  t=0  R  0 

A  policy  R*  is  said  to  be  average-cost  optimal  if 

Cp(Z,R*)  =  min  cp(Z,R)  for  all  states  Z. 

R 

We  shall  need  the  following: 

Condition  4 1 ;  P  >  0 

Lemma  7 •  3 •'  Under  Conditions  1'  ,2, 3>4',  for  any  (stationary)  policy  R, 
Mzq ( R )  <  N/P1q  for  all  states  Z-=0,  (n,i)  i  >  0,  n  <  N-l; 
where  M7/^(R)  denotes  the  mean  recurrence  time  to  go  from 
state  Z  to  state  0  when  policy  R  is  employed. 


Proof:  Let  =  number  of  units  in  reserve  at  time  t.  Then  since 

P.n  >  Pir,  for  all  i  >  0  it  follows  that  P_(Y.  .  =  n-l|Y,  -  ni  >  P.n 
iO  —  10  R  t+1  1  t  -  10 

for  any  policy  R.  The  lemma  follows  immediately.  Q.E.D. 


Theorem  7  -4 :  Under  Conditions  1', 2,3,4'  there  exists  bounded  numbers 
g,  f(n,i)  n  =  0,1,..., N-l,  i  >  0  such  that 

(i)  (f(n,i)  -  C(n,i)j”_^  is  non-decreasing  for  each  n=0,l, . . . ,N-1 


(il)  (a) 
(b) 


g  =  min  (C(K)  +  E  P.  f(K-l, j) } 
l<K<n  j=l  J 


g  +  f (n, i)  =  C(n, i)  +  min  f  E  P±  f(n, j)  +  PiQ 


0 


E  P,  f(n-l, J ) 


J=1 


'j 


(where  f(-l,j)  =  0  for  all  j) 


E  Pf(n-l,j) 
J»1  J 
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(ii)  (c)  The  policy  R*  which  when  in  state  (n,i)  takes  action 
1  when  the  first  term  of  (b)  is  minimum  and  action  0  other¬ 
wise,  and  which  when  in  state  0  selects  the  action  which 
minimizes  (a)  is  average-cost  optimal. 

(iii)  qp(Z,R*)  =  g  for  all  states  Z. 

Proof:  The  proof  of  (ii)  and  (iii)  follows  from  Lemma  7*3  and  Theorems 
1.1,1.2,  and  1.4  of  [5].  (i)  follows  from  Lemma  7*1  and  the  remark 

following  the  proof  of  Theorem  1.1  of  [5]-  Q.E.D. 

Corollary  7 .  5-'  Under  Conditions  l',2,3,L'  there  is  a  generalized 
control-limit  policy  which  is  average-cost  optimal. 

Proof:  From  (ii)  of  Theorem  7.4  the  policy  which  replaces  when  in 
state  (n,i)  iff 

00 

1/1-P  E  P  f(n, j)  >  E  F  f (n-1, J ) 
j/0  j=l  J 

is  average-cost  optimal.  The  result  then  follows  from  part  (i)  of 
Theorem  7-4  and  Lemma  2.1.  Q.E.D. 

We  now  prove  the  analogues  of  Theorems  4.3  and  4.4 

Theorem  7»6:  Under  Conditions  l',2,3,4'  if  (n,i)  is  such  that 

l/l-P.„  E  P  C(n,j)  <  g  then  there  is  an  average-cost  optimal 
j/0  1J 

generalized  control-limit  policy  which  does  not  replace  at  state  (n, i). 

Proof:  From  (ii)  of  Theorem  7*4  we  have  that 

00 

g  +  f (n, j  )  <  C( n, j )  +  E  P  f (n-1, j ) 

J=1  J 

00 

.‘.g  +  l/l-P  E  P  f(n,j)  <  1/1-P  E  P  C(n,j)  +  E  P  f(n-l,j) 
1  j/0  10  1J  J=1  J 
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1/1-P.n  s  P,  .  C(n,j)  <  g  =>  l/l-P.n  £  P. ,f(n-l,j) <  £  P.f(n-l,j) 
10  j/0  iJ  “  J/O  j=l  J 

The  Theorem  follows  from  Corollary  7-5  Q.E.D. 


Theorem  7*7  ■  Under  Conditions  1' if  F  =  0  for  all  0  <  j  <  i 

^  J 

(pure  deterioration)  then  the  generalized  control-limit  policy  which 
replaces  at  state  (n,l)  iff 

l/l-P,r.  £  F,  ,  C(n,j)  g  is  average-cos',  optimal. 

JA>  1J 

Proof:  Suppose  that  l/l-P.  .  £  P  ,  C(n,j)  >  g  and  suppose  further 

iu  \/.Q 

that 

X 

l/l-P  £  P  f(n,j)  <  £  P,  f ( n - 1 , j ) 

°  j/0  IJ  j=l  J 

CO 

r  +  f(n,i)  =  C(n,i)  *  £  P  f (n, J )  +  P  £  P  f(n-l,j) 

,1/c  1J  10  j*l  J 

=  c(n,i)  -  £  P..  C(n,j)  *  £  P.,(f(n,j)  -  C(n,j)) 
J^o  1J  ita  1J 

X 

+  P,n  s  P.  f(n-l,j) 

J=1  J 

>C(n,i)  -  g(i-Pi0)  t  (l-c.0)(f(n,i)  -  C(n, i) ) 


+  P  £  ?  f(n-l,j) 

j-1  J 

Ot 

•*•  8  +  f(n,i)  >  C(n,i)  *  £  F  f(n-l,j) 

,1=1  J 

which  is  a  contradiction  by  Theorem  7  .b  (ii) 


l/l-P, n  £  P,  ,  f(n,j)  >  £  F.  f(n-l,j) 
j/0  10  j=l  J 

'The  result  follows  from  Corollary  7. 5  and  Theorem  7-6.  Q.E.D. 
Corollary  7 .Q:  Under  the  conditions  of  Theorem  7-7  if  C(n,i)  is  monotone 
non-decreasing  in  n  for  i  fixed  then  there  is  an  optimal 


generalized  control-limit  policy  such  that  i  ^ 


2  b 


Proof:  Follows  from  Theorem  1.1. 
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